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Abstract. - Recently it was argued that quantum phase transitions can be radically different 
from classical phase transitions with as a highlight the 'deconfined critical points' exhibiting 
fractionalization of quantum numbers due to Berry phase effects. Such transitions are supposed 
to occur in frustrated ('J1-J2') quantum magnets. We have developed a novel renormalization 
approach for such systems which is fully respecting the underlying lattice structure. According 
to our findings, another profound phenomenon is around the corner: a fluctuation induced 
(order-out-of-disorder) first order transition. This has to occur for large spin and we conjec- 
ture that it is responsible for the weakly first order behavior recently observed in numerical 
simulations for frustrated 5=1/2 systems. 



Introduction. - Frustrated magnets exhibit quantum phase transitions of a rich variety 
which is subject of intense current research [1]. Novel scenarios for phase transitions beyond 
the Landau- Ginzburg- Wilson (LGW) paradigm have been suggested [2,3] joggling fundamen- 
tal concepts. The Heisenberg model on a square lattice with antiferromagnetic couplings J\ 
and J2 between nearest and next-nearest neighbors serves as a prototype for studying mag- 
netic quantum-phase transitions (see, e.g., [4] and references therein). From the classical limit 
one expects that two different magnetic orders can exist: the Neel phase with ordering wave 
vector (ir, w) is favorable for a = J2/J1 < 1/2 and columnar order with (0, it) for a > 1/2. 

Quantum fluctuations certainly may induce a paramagnetic (PM) phase which is naturally 
expected near a ~ 1/2 where both orders compete [5]. Remarkably, additional orders may 
appear in the Neel phase as well as in the PM phase when translation symmetry is broken by 
an additional spin dimerization [4]. The existence of such enhanced order crucially depends 
on the spin value S. This becomes most apparent when the spin system is represented by a 
nonlincar-sigma model. Topological excitations associated with Berry phases can give rise to 
ground-state degeneracies corresponding to a translation-symmetry breaking by dimerization 
and formation of valence-bond solid (VBS) phases [6,7]. 
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The Neel-VBS transition has been argued to be in a novel quantum criticality class that 
does not fit in the standard LGW paradigm. Intriguing data on this transition was obtained 
in simulations of the 5 = 1/2 quantum XY model frustrated by ring exchange [8,9]. The 
transition was interpreted as a second-order one; this possibility was predicted by the theory 
of the deconfined critical point and suggested to be generic for a variety of experimentally 
relevant two-dimensional antiferromagnets [3]. However, in a more careful finite-size analysis 
of the XY case it was demonstrated that the Neel-VBS point represents an anomalously weak 
first-order transition [10]. 

Field-theoretical approaches of various kinds have been developed, based on the 1/5 [11] 
or the 1/N expansion [12,13]. The latter approach is able to capture some of the essence of the 
topological aspects on a mean-field level. However, it is the former approach, elaborated to a 
renormalization-group analysis, which predominantly has served as basis for a comparison of 
critical aspects between theory and experiment (see e.g. [14]). 

Nevertheless, this approach so far has suffered from two intrinsic shortcomings: i) Spin- 
wave interactions are the physical mechanism underlying the renormalization flow. On one- 
loop level, the flow equations describe corrections to the physical parameters of relative order 
1/5. These corrections have been worked out using a continuum version of the nonlinear 
a model (CNLctM) as a starting point [11]. However, for the original lattice model, this 
is not systematic, since corrections of the same order are dropped under the naive coarse 
graining of the lattice model onto the CNLctM. As a consequence, important effects such as 
a renormalization of the spin-wave velocity and the frustration a are missed, ii) Similarly, 
the outer large momentum region of the magnetic Brillouin zone (BZ) is only crudely treated. 
This entails a significant uncertainty in the computation of the phase diagram. 

In this Letter, we lift these shortcomings by developing a renormalization analysis which 
fully accounts for the lattice structure. It combines the systematic treatment of all corrections 
in order 1/5 on the level of the conventional first-order spin- wave theory (SWT) with the 
merits of a renormalization approach, which goes beyond any finite order in 1/5 by an infinite 
iteration of differential steps, successively eliminating the spin-wave modes of highest energy. 
As a result, we obtain an improved description of the phase transitions. In particular, the 
possibility of a fluctuation induced first-order transition which is not accessible on the level 
of the CNLctM is included in a natural way. 

Spin- coherent state representation. - The key to what follows is a novel kind of path 
integral quantization, which makes it possible for us to treat the effects of umklapp on an 
equal footing with the spi-wave interactions. To be specific, we stick to the aforementioned 
J\- J2 Heisenberg model on a square lattice. We address the stability of the Neel phase against 
quantum fluctuations controlled by 5 and a. Fluctuations are treated in a coherent spin state 
path-integral representation of the model, where a spin state corresponds to a unit vector n. 
In the absence of fluctuations, spins would assume the states Iua/b) — I ± e z ) = |5, ±5) on 
the two sublattice A and B. From the standard Trotter formula emerges an imaginary time r 
(discretized in intervals of duration At) and an action of the form [1] 

taking n at time r and n' at r — At. For weak quantum fluctuations, the components of 
n x,y perpendicular to the magnetization axis are small and may be considered as expansion 
parameters (as in [11]). However, attempting to directly apply this expansion to the lattice 
model we encountered time ordering difficulties in the action [15]. 
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Instead, we start form a stereographic parametrization of coherent states, on sublattice A, 
|n) = (1 + aa/25) _s exp(a5_/\/25)|5, 5) where a is the stereographic projection of n from 
the unit sphere onto the complex plane, a — tan(0/2) exp(i^) with the standard spherical 
angles 9 and <p. The action can be expressed in terms of the stereographic coordinates using 
the matrix elements [16] 

(n\S z \n'} = l-aa'/2S (n\S+\n') = y/2S a' 
(n\H') l + aa'/2S" (n|n') l + aa'/2S { ' 

Here a is the complex conjugate of a. The expressions for the coordinate b on sublattice B 
are given by the correspondences S z — > — 5 Z and S± = S x ± S y — > 5 T for a — * 6. 

The explicit expression of the action in terms of a and b is too lengthy to be given here. To 
leading order in 1/5, the fluctuations are controlled by the bilinear part Sq of the action that 
represents free magnons. We also retain the quartic contribution <S;„t to the action, which 
represents magnon interactions and contain the renormalization of single-magnon parameters 
of relative order 1/5. Higher order contributions to the action are neglected. Terms from the 
functional Jacobian are also negligible on this level. The single-magnon contribution to the 
action can be parameterized in the form 

Sq = / {^[a k Aa k - Aa k a k + 6 k A6 k - A6 k & k ] 

+Ar5[r+(a k a k + 6 k 6 k ) + T k (a k 6 k + a k 6 k )]}, (3) 

using the Fourier transform a(r) = J k e ,k r a k , the intgeral J k = 2 J £Jk over the magnetic 

BZ \k x \ + \k y \ < 7T, and the exchange couplings T k = j£ — Jq +Jq, T k = J k = 2Ji[cos(fca;) + 
cos(fc w )] and J k = 2J 2 [cos(fc K + k y ) + cos(k x — k y )]. For simplicity the lattice constant is 
considered as unit length. The dimcnsionlcss parameter g represents the strength of quantum 
fluctuations. It assumes the value g = 1 in the unrenormalized model and turns out to increase 
under renormalization. 

Diagonalizing this bilinear action, one easily obtains the magnon dispersion £" k = 5g|r k | 
yjl — 7 k , where 7 k = r^/F^. For a < 1/2, the low-energy spin-wave excitations are char- 
acterized by an isotropic dispersion E(k) = c|k| + 0(k 2 ) with a spin-wave velocity c = 
V^gSJiy^l — 2a. Likewise, the exchange couplings generate a spin stiffness p = 5 2 Ji(l - 2a) 
for low-energy modes. One also obtains the propagators (0 k k ) = gG k and (</> k k ) = 
(0 k 0k) = ff(G k + l) for fields from the same sublattice {<f> = a, 6), and (a k 6 k ) = (a k 6 k ) = —gF^ 
for fields from different sublattices. In the latter case the correlators are unchanged by 
a replacement 4> — > 4>' . We have defined G k = (n k + 1/2)(1 — 7 k ) -1 ^ 2 — 1/2 and F k = 
(n k + l/2)7t(l — 7 k ) -1 ^ 2 where n k = (e^ Ek — 1) _1 is the bosonic occupation number. For 
strong frustration (a > 1/2) the stiffness becomes negative and the spin- wave velocity is ill 
defined due to the presence of unstable modes in the center of the BZ (see Fig. QJ. 

Renormalization approach. - Starting from this action with bilinear and quartic terms, 
we implement a renormalization procedure as follows. In successive steps, the modes of highest 
energy (an infinitesimal fraction of all modes) are integrated out. This decimation of modes 
yields an effective theory for the remaining modes and gives rise to differential flow equations. 
As flow parameter we choose I = h ^(^bz/^rbz), where Abz = 2tt 2 is the area of the original 
BZ, and Arbz is the area of the residual BZ (RBZ) populated by the remaining modes. On 
large length scales in the Neel phase, the RBZ becomes circular and / is the usual logarithmic 
length scale. The evolution of the RBZ and the single-magnon dispersion is illustrated in 
Fig. H for various parameters. 
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Fig. 1 - Evolution of modes under coarse graining. Each panel corresponds to the area \k x \ < vr, 
\k y \ < 7T. Red color represents small, yellow high positive energy. Black lines are lines of constant 
energy, blue areas represent unstable modes. Panels (a)-(d): In the Neel phase the RBZ may become 
disconnected first, then always shrinks to a sphere (here S = 1/2, a = 0.3, / = 0,0.25,0.5,1.0). 
Panels (e)-(h): In the Neel phase for strong frustration a > 1/2, initially unstable modes (blue area) 
are renormalized to stable modes and the RBZ eventually also shrinks to a sphere (here S = 1/2, 
a = 0.55, / = 0,0.11, 0.29, 1.30). Panels (i)-(k): In the columnar phase, after the elimination of all 
stable modes, an area of unstable modes survives (here S — 2, a = 0.6, 1 = 0, 0.10, 0.20). 



To one-loop order, corresponding to a systematical calculation of corrections in order 1/S, 
the renormalization effects due to spin-wave interactions can be captured by a flow of the 
single-magnon parameters. The resulting flow equations are given by 

dg- 1 = ~dG°, (4) 
dJi = |ji(dF- -2dG°), (5) 
dJ 2 = |j 2 (dG+-2dG°), (6) 

where the integrals over the differential fraction d of modes of highest energy are defined 
as dG° = J q a G q , dG+ = J q 9 (J+/ J+)G q , and dF~ = f^J-/J^)F 9 . _ 

Since the BZ does not remain self-similar under mode decimation, we omit the usual 
rescaling of length and time which is necessary only for the identification of fixed points under 
a renormalization-group flow. However, dropping this rescaling does not lead to a loss of 
information. Then, each fixed point represents an antiferromagnetically ordered state. The 
quantum-disordered phase and the transition into it show up as run-away flow. 

Results and discussion. - Because of the changing geometry of the RBZ and the incorpo- 
ration of the full single-magnon dispersion in our renormalization approach, the flow equations 
can be integrated only numerically. Here, we focus on T = 0, although the flow equations 
are valid also for T > 0. The flow of parameters is characterized by the following tendencies. 
Both exchange couplings J1.2 as well as 1/g always shrink. These fundamental parameters 
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flow in such a way that a always decreases, c 2 increases (initially it is negative for a > 1/2), 
while p may increase for small / until it decreases for sufficiently large I. 

The nature of magnetic order can be identified from the flow behavior. Three possibilities 
are observed, (i) The RBZ shrinks to a circle of decreasing radius oc e~ l [see Fig. ^ panels 
(a)-(d) and panels (e)-(h)] while and g (as well as the derived quantities c, p, a) converge 
to positive values. Then Neel order is present, characterized by these renormalized low-energy 
parameters, (ii) At some finite I*, fluctuations become so strong that g diverges and 
vanish. This indicates the loss of magnetic order due to overwhelming quantum fluctuations. 
Close to the transition to the Neel phase, the magnetic correlation length - which can be 
identified with £ = e l - diverges algebraically like £ ~ (a — Q! c ) — 1 . For S = 1/2 this 
asymptotic behavior is shown in one inset of Fig. |5J (iii) For strong frustration, it is also 
possible that a finite RBZ of unstable modes remains after decimation of all stable modes (see 
Fig. ^panels (i)-(k)). This indicates that the instability towards columnar order is effective 
for the renormalized low-energy modes. 




Fig. 2 - Phase diagram showing the 2nd order transition line between the Neel ordered and the PM 
phase (dashed line) and the 1st order boundary between the Neel and the columnar phase (solid 
line). The border between the PM and the columnar phase (dotted line) is just guesswork and 
cannot be calculated within our approach. Insets: Nature of the phase transitions. For 5 = 1/2 
the renormalized spin stiffness p* vanishes and the strength of the quantum fluctuations Z*^ diverges 
at the phase boundary corresponding to a 2nd order transition. Close to the the Neel phase the 
correlation length diverges like £ ~ (a — a c ) ■ At the 1st order transition (S = 2) p* and Z^ remain 
finite. 



The region of stability of the Neel phase is illustrated by the light grey region in Fig. [21 
In the absence of frustration, we find Neel order to be stable for 1/S < 5.09 in remarkable 
agreement with conventional linear SWT [5]. However, this is pure coincidence since in linear 
SWT the phase boundary is determined by the vanishing of the local magnetization calculated 
in order S° , whereas here it is determined by the divergence of g due to spin- wave interactions 
treated in one-loop order. While the phase boundary is located at academically small spin 
values for small frustration, S reaches physically meaningful values at larger frustration where 
the discrepancies between SWT and our renormalization approach become more pronounced. 
In linear SWT, the phase boundary smoothly approaches 1/S \ for a / 0.5, whereas we 
find the Neel phase to reach up to a — 0.66 for S — 0.68. For spins smaller than this value, the 
Neel phase becomes unstable towards a PM phase via a second-order transition, whereas it 
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becomes unstable for 5 > 0.68 via a first order transition. Since we deal with a discontinuous 
transition we can only speculate about the type of ordering in the adjacent phase (dark shaded 
region in Fig. |5J) . From the classical limit we expect columnar order for very large 5 whereas 
for intermediate spin also VBS order may be present. 

In the region where the Neel phase reaches up to a > 1/2, initially unstable modes are 
renormalized to stable ones by spin- wave interactions. Simultaneously, a is renormalized to 
a value a* < 1/2 and the flow behavior (i) is realized. In the columnar phase, the flow of a 
saturates at a value a* > 1/2 and the flow behavior (iii) is observed. 

Stability of Neel order for a > 0.5 so far has been found only by a self consistently modified 
SWT [17] and Schwinger-boson mean-field theory (SBMFT) [18]. The overall shape of the 
Neel phase of these approaches is consistent with our findings. However, in modified SWT 
and SBMFT the first-order transition from Neel to columnar order can be obtained only by 
a comparison of free energies between the two phases. In our theory, the transition directly 
emerges from the analysis of spin-wave interactions in the Neel phase. 

The nature of the transitions out of the Neel phase becomes apparent from the behavior of 
the fluctuations on large length scales (k — ► 0), where (<^k^k) — with = ^ l 9 _ 2a oc |. 
Approaching the transition into the PM phase, the renormalized value diverges and gives 
rise to a divergent susceptibility (see Fig. |3J) . At the same time the renormalized p* vanishes 
while c* remains finite. The continuous evolution of Z^ and p* indicate the second-order 
nature of the transition. Approaching the transition into the columnar phase, one observes 
a saturation of Z£ and p* at finite values as well as a discontinuous jump of a* indicating a 
first-order transition. Fig. [3] illustrates the dependence of various renormalized quantities on 
a and 1/5 within the Neel phase. The insets of Fig. 2 show the evolution of p* , Z^ and £ 
with higher resolution at the transitions. 






0-1 0.2 0.3 0.4 0.5 0.6 



0.1 0.2 0.3 0.4 0.5 0.6 0.7 



Fig. 3 - Values of the renormalized parameters within the Neel phase. Color corresponds to the 
strength of large-scale fluctuations (a), the renormalized spin stiffness (b), the frustration (c), and 
the spin- wave velocity (d). 



Confidence in the reliability of our findings is provided by quantitative comparisons for 
specific parameter values. Results for Z* = c* /c exist from various approaches. For 5=1/2 
and a = 0, first-order SWT yields a slight enhancement of spin- wave velocity, Zf WT = 
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1.158. We find an increased value Z* c = 1.20, which is in agreement with Monte Carlo (MC) 
simulations (see [19] and references therein). As 1/5 and/or a increases, the enhancement 
factor Z* also increases. At the phase boundary 1/5 w 5.09 for a = 0, the difference is already 
more pronounced: Zf WT = 1.40 and Z* = 2.04. For a > 0, unfortunately, MC data for Z c 
are not available at the transition out of the Neel phase, neither for 5 = 1/2 nor for larger 
physical values of 5. 

Conclusion. - We have presented a novel renormalization approach for frustrated quan- 
tum antiferromagnets which fully accounts for the underlying lattice geometry and consistently 
captures the renormalization of the single-magnon parameters by spin-wave interactions all 
over the magnetic BZ. 

For the J\-Ji model we clearly demonstrated that for large spins and strong frustration, 
fluctuations on lattice and intermediate scales cause an instability of the Neel phase towards 
a first order transition. These effects are totally missed by any effective long-wavelength 
continuum theory obtained by a naive coarse graining of the lattice model. 

We conjecture that these fluctuations which crucially depend on the underlying lattice 
geometry and the way of frustrating the Neel order are responsible for the weakly first order 
behavior observed recently in numerical simulations for frustrated S = 1/2 systems [10]. 
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